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Abstract 

We present a study of the decays 5° ]ji*)Oj{{*)o yging a sample of 124 mhhon T{AS) BB 
decays collected with the BABAR detector at the PEP-II asymmetric-energy e~^e~ collider at SLAC. 
We report evidence for the decay of B° and B^ mesons to the D*^Kg final state with an average 
branching fraction B{B D*^K^) = (4.5 ± 1.9 ± 0.5) x 10~^. Similarly, we measure B{B 
D^K^) = (6.2 ± 1.2 ± 0.4) X 10"^ for the D^K^g final state. We also measure i3(B° ^ D^K*^) = 
(6.2 ± 1.4 ± 0.6) X 10-5 and set a 90% C.L. upper limit B{B^ D^K*^) < 4.1 x 10"^ Ah results 
presented in this paper are preliminary. 
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With the discovery of CP violation in the decays of neutral B mesons 1 and the precise 
measurement |2j of the angle (5 of the Cabibbo-Kobayashi-Maskawa (CKM) Unitarity Triangle [3], 
the experimental focus has shifted toward the measurements of the angles a and 7. Several methods 
have been suggested to measure 7 with small uncertainties, but they all require large samples of B 
mesons not yet available. The decay modes B^ — > D'^*^^K^ offer a new approach for determination 
of sin(2/3 + 7) from measurement of time-dependent CP asymmetries in these decays The 
CP asymmetry appears as a result of interference between two possible diagrams leading to the 
same final state D^*^^K^ (Figure P): a 6 ^ c transition B^ Di*)Oj{0 ^^^^ 5 _> ^ transition 

_^ i)(*)o^o_ A ^0 meson can either decay to a D^*'>^K° final state, or oscillate into 

a B^ which then decays to a D^*'^°K° {D^*^'^K'^) final state. 

The B^B^ oscillation provides the weak phase 2/3 and the relative phase of the two decay 
diagrams is 7. 




d d 



Figure 1: The decay diagrams for the b ^ c transition B^ ^.j^g 6 ^ u transition 

The sensitivity of this method depends on the ratio r = \A{B^ D'^*^^K^)\/\A{B° 
£)(*)o^o^| q£ ^j^g decay amplitudes. The ratio of the CKM matrix elements in the two am- 
plitudes naively suggests r ~ 0.4, however, the B decay dynamics can modify this expectation. 
The ratio r can be probed by measuring the rate for the decays B^ D^K*^ and B^ D^K*^, 
using the self-tagging decay K*^ K~-k~^ . 

The B^ — > D^K*^ and B^ D^K*^ decays are separated by means of the correlation between 
the charge of the kaons produced in the and K*^ decays: in the former decay the two kaons must 
have the same charge, while in the latter they are oppositely charged. In the B^ — > D^*^^K^ decays, 
the strangeness content of the K'^ is hidden and one cannot distinguish between B^ — > D^*^^K^ and 
B^ — > D^*^^K^ . Hence in the remainder of this paper we refer to these decays as B ^ D^*^^K^. We 
estimate the branching fractions B{B i)(*)0/f{*)0) from the measured color-suppressed decays 
B'^ — > D^*^^7r^ [3] to be approximately 3 x 10^^. In this paper, we present the first evidence for the 
decay B D*^K^, measurements of the branching fractions B{B D^K'^) and 5(5° D^K*^), 
and a 90% C.L. upper limit for the branching fraction of the b ^ u transition B^ D^K*^ . 

Results presented here are based on a sample of 124 million T(45) — > BB decays collected 
with the BABMi detector between 1999 and 2003 at the PEP-II asymmetric-energy e^e~ collider 
operating near the T(4S') resonance. The properties of the continuum e"^e~ ^ qq (q = u,d,s,c) 
events are studied with a data sample of 12 fb~^ recorded 40 MeV below the T{4S) resonance. 
We also use large samples of simulated T(45) BB and e"*"e~ — > qq events which are about three 
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and 15 times the size of the data, respectively. 

The BABAR detector is described elsewhere [H]. Only detector components relevant for this 
analysis are summarized here. Trajectories of charged particles are measured in a spectrometer 
consisting of a five-layer silicon vertex tracker (SVT) and a 40-layer drift chamber (DCH) operating 
in a 1.5 T axial magnetic field. Charged particles are identified as pions or kaons using information 
from a detector of internally reflected Cherenkov light, as well as measurements of energy loss 
from ionization {dE/dx) in the SVT and the DCH. Photons are detected using an electromagnetic 
calorimeter (EMC) constructed of 6580 thallium-doped Csl crystals. 

We reconstruct the decays D^K^, D*^K^, D^K*^, and D°K*° in the decay chains: 

D*° D^-K^; D° K-Tr+, R-tt+tt'^, and K-tt+tt-tt+; ^ ^ vr+vr-; K*^ K+ir-; 
and vr" 77 (throughout this paper charge-conjugated decay modes are implied unless explicitly 
specified). For each decay channel, the optimal selection criteria are determined by maximizing 
the ratio S/\/ S + B, where S and B are, respectively, the estimated signal and background yields 
in simulated events. A large sample of the more abundant B~^ — > D^7r~^ decays, in which the 
decays to the K^tt^, K^tt^tt^, and K^tt^tt^tt^ final states, is used as a calibration sample to 
measure efficiencies and resolutions for the selection variables. 

Charged tracks used to reconstruct and K*^ candidates are required to have transverse 
momentum px > 100 MeV/c, and the candidates must satisfy kaon identification criteria. These 
identification criteria have an average efficiency of about 90% while the probability of pions being 
mis-identified as kaons varies between a few percent and 15%. The photons are reconstructed from 
clusters in the electromagnetic calorimeter with energy greater than 30 MeV that are consistent with 
photon showers. We select ir^ candidates from pairs of photon candidates and require 115 MeV/c^ < 
m(77) < 150 MeV/c^. 

The Kg candidates are selected from pairs of oppositely charged tracks with invariant mass 
within 7 MeV/c^ {2a) of the nominal mass [Tj. The displacement Ij^o of the decay vertex 
from the interaction point in the plane perpendicular to the beam axis divided by its measured 
uncertainty fj^^o must be greater than 2. The K*^ candidates are selected from pairs of oppositely 
charged K'^ and tt~ tracks, with invariant mass m{K^7r^) within 50 MeV/c^ of the nominal K*^ 
mass j7). We also require | cos6h\ > 0.4, where 9^ is the K*^ helicity angle, defined as the angle 
between the direction of the K*^ in the i3^-meson rest frame and the direction of its daughter 
in the K*^ rest frame. For signal B^ D^K*^ candidates. Oh follows a cos'^ Oh distribution while 
the combinatorial background is distributed uniformly. 

We reconstruct the candidates in the K~tt~^ and K~ it~^ t:~ t:~^ decay modes by combining 
charged tracks. Combinations with an invariant mass within 2a of the nominal mass m^o are 
retained. In the K'tt^tt^ selection, the tt^ candidates are required to have a center-of- 

mass (CM) momentum p*o greater than 400 MeV/c. For each K^-k^tt^ combination, we use the 
kinematics of the decay products and the known properties of the Dalitz plot for this decay [HI 
to compute the square of the decay amplitude \A'\^. We select combinations with \A\^ greater 
than 5% of its maximum value. This requirement selects mostly the Kp region of the Dalitz 
plot. It rejects 62% of the combinatorial background and has an efficiency of 76%, as measured 
with the B~^ — > D^tt"*" calibration sample. The combinations with the invariant mass within 
25 MeV/c^ (2.5fj) of m^o are retained. For the purpose of cross checks, we define a mass 
sideband as the region between 45 and 90 MeV/c^ away from m^^o in the K~-k~^ and 

K^-K^TT^-K^ mode, and between 85 and 160 MeV/c^ away from m£)0 in K^tt^tt^. 

The D*^ candidates are selected from combinations of a and a ir^ with p*o > 70 MeV/c. 
After kinematically constraining and candidates to their nominal masses, we select the 
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candidates with the mass difference Am = |m(D*°) - m{D^) - 142.2 MeV/c^| < 3.3 MeV/c^. 
Two standard kinematic variables are used to select candidates: the energy-substituted 



mass ruEs = y {^s + po • pb)"^ / E'^ — p% and the energy difference AE = E*^ — \^/s^ where the 
asterisk denotes the CM frame, s is the square of the total energy in the CM frame, p and E 
are, respectively, three-momentum and energy, and the subscripts and B refer to T{AS) and B^ , 
respectively. For signal events, m-Es is centered around the B^ mass with an r.m.s. resolution 
of about 2.6 MeV/c^, dominated by the knowledge of the and e~ beam energies. The AE 
resolution is dominated by the momentum and energy resolutions of the detector and hence varies 
for different decay modes. We constrain the mass of the D^*)*^ and candidates to their respective 
nominal values. In simulated events the r.m.s. AE resolution is found to be ~ 13 MeV for all 
B^ decay modes. The B^ candidates are required to have AE within 30 MeV of the mean value 
measured in the B^ D^ir^ calibration sample. 

We use two variables to reject most of the remaining background, which is dominated by contin- 
uum events: the polar angle of the B'^ candidate in the CM frame and a Fisher discriminant [01 
based on the energy flow in the rest of the event, after removing the B^ decay products. The Fisher 
discriminant is defined as a linear combination of | cos \ , where is the angle in the CM frame 
between the thrust axis of the B^ and that of the remaining charged and neutral particles in the 
rest of the event, and the Legendre monomials and defined as = YljPj cos*0j; here Pj 
is the CM momentum and 9j the angle between the direction of remaining particles in the event 
with respect to the thrust axis of the B^ candidate. The requirement on varies for each decay 
channel because of different levels of the background. In the D^*^^Kg and D^K*^ final states our 
requirement has a typical signal efficiency of about 80% while rejecting about 85% of the back- 
ground. A tighter requirement in the B^ — > D^K*^ mode rejects 95% of the background and has 
a signal efficency of 55%. The requirements on angle 0^ are | cos^^| < 0.75 for B^ D^K*^, and 
I cos^IjI < 0.85 for all other decay modes. 

In about 2-3% of the events more than one B^ candidate with niEs > 5.2 GeV/c^ satisfy the 
selection criteria. In the D^K^, D^K*^ , and D^K*^ final states the candidate with the smallest AE 
is selected. In the D*^K^ final state, we pick the candidate with the smallest computed from 
the measured value of m[D^) and m{D*^) — m[D^), their nominal values, and their resolutions in 
data. 

The signal yield for each B^ decay mode is determined with a binned maximum likelihood fit 
to the niES distribution for each decay mode. The distribution is modeled with a Gaussian for 
the signal and a threshold function for the combinatorial background. The mean and the r.m.s. 
resolution of the Gaussian are fixed to values measured in the B~^ — > D^tt^ calibration sample. The 
threshold function parameterizing the background is defined as A{mEs) ~ "iesVI — x'^ exp{— (^(1 — 
x^)} ^0]) where x = 2mEs/\/s and is a shape parameter. 

The measured signal yields are summarized in Tabled and the ttt-es distributions for the sums 
of all three decay modes are illustrated in Figure |2j The signal yields measured from the AE 
distribution, after removing the requirement on AE and selecting candidates with 5.273 GeV/c^ < 
"lES < 5.288 GeV/c^, are found to be in good agreement with the uies yields. In this case, for the 
small fraction of events with more than one selected candidates, we choose the one with mES closest 
to the B mass. The AE distribution for the signal is modeled with a Gaussian and a second-order 
polynomial is used for the background. The AE distributions for the selected events are also shown 
in Figure 121 The combinatorial background in both m-Es and AE distributions are described well 
by events in the sidebands of the mass, which are shown as hatched histograms in Figure (21 As 
a further cross check we examine the helicity angle 9h of the B^ D^K*° candidates with 
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Table 1: Signal yield Ns, estimated peaking background N"^^, effective signal efficiency eeffi and 
the measured branching fraction B for the B — > decays. The efficiency Seflf is defined 

as e X BF, where e is the signal reconstruction efficiency and BF are the appropriate intermediate 
branching fractions for D*^ , D^, K*^ , and decays to final states reconstructed in this analysis. 



5.273 GeV/c^ < rriEs < 5.288 GeV/c^ after removing the requirement \ cosOh\ > 0.4. Figure |31 
shows the distribution of cos 9h for these candidates, after subtracting the combinatorial background 
from the sidebands. We fit this spectrum with a sum of a flat component for the background 
and a cos^ 6h distribution for the signal, and find the fraction of the former to be consistent with 
zero as expected. 

The measured signal yields Ns include a small contribution from other B decays that can be 
mis-reconstructed as signal events and accumulate near the B mass. We have studied the contribu- 
tion of such potential "peaking" backgrounds with large samples of simulated events, corresponding 
to typically between 100 and 1000 times the size of our data, for the following categories of decays: 
(1) B^ — > D^p^,p^ — > 7r~^7r~ decays, where one of the two pions is mis-identified as a charged 
kaon; (2) the B^ — > D^n^ decays, followed by the Cabibbo-suppressed decays D~ K^*^^K~ 
and 5° D-K+, K^*^^tt- reconstructed in the D^{K-tt+)K^*^^ final states; (3) charm- 

less B^ — > K~^'K~K^{m:) where the and vr" are wrongly combined to form a K^it~ 
candidate; (4) B° D*°K^*^°, D*^ D°-f candidates, where a low -energy photon 7 is not re- 
constructed; (5) the decays B+ D*^K+,D*^ ^°7r°/7, D^K*+,K*+ K+TT^,K^Tr+, 
and B^ — > D*~ , D*^ D^ir^, where a low-energy vr" or photon is replaced by a random low- 
momentum charged track. The contribution of category (1) is found to be less than 0.01 events 
and is hence neglected. The contribution of category (2) is also negligible in all modes, except for 
B — > D^K^,D^ K^TT^. We eliminate 87% of these events by requiring the invariant masses 
m{KgK~) and m^Kgir^) to be more than 20 MeV/c^ away from the nominal mass. The tties 
spectrum of the remaining background events in this category and in categories (3)-(5) shows a 



11 



broad enhancement near the B mass. The contributions of these events to the signal yields are 
measured by performing a Monte Carlo study and are summarized in Table ^ We assign a 50% 
systematic error to the estimated peaking background contribution due to the uncertainty on the 
branching fractions of some of these B decays. In the decay B^ — > D^K*'^, the charge correlation 
used in the selection removes all contributions from known B decays included in our simulation. 
We estimate the peaking background for this decay mode from the sidebands to be 5.5 it 4.5 
events. 

The significance of the signal yields is determined by taking into account the sum Nb of the com- 
binatorial and peaking backgrounds and its uncertainty 5Nb- We generate one million experiments 
where the expected number of background for the i-th experiment is extracted from a Gaussian 
with mean Nb and width 5Nb- Assuming a Poisson distribution for the background events, the 
significance is determined from the fraction of experiments with N^^ > A'obs with A'obs being the 
number of candidates in data with 5.273 GeV/cP < ruES < 5.288 GeV/c^. The B D*^K^ signal 
has a significance of 3.3cr and is the first evidence for this decay mode. The significance of the 
B^ D^K*^ and B^ D^K*° are, respectively, 4.8cr and 2.0a. For the D^K'^ final state, where 
the number of observed events is large, we estimate the signal significance to be 5.8cr from the 
measured signal yield and its uncertainty. 

The systematic uncertainties for the branching fractions are reported in Table ^ and include 
contributions from estimated peaking background (3-11%), fit parameters (2-8%), D^*^^ branching 
fraction (2.4-6.9%), vr*^ reconstruction efficiency (2.5% per photon), charged-track reconstruction 
efficiency (0.8% per track), Monte Carlo statistics (1-4%), efficiency correction factors (1-4%), kaon 
identification (2% per kaon), Kg reconstruction efficiency (1.6%), and number oiBB events (1.1%). 

The branching fraction B of each B^ decay mode is computed as the weighted average of the 
branching fractions Bj in each channel Dj = {i^~7r^, X~7r^7r^7r^, i^^vr^vr^}, computed as 

^ Njjg X Bd, X Bk X Ej 

where Ns^ is the signal yield from the mES fit) is the estimated peaking background from 
Tabled Nbb is the total number of T(4S') BB events, B^j is the branching fraction B{D^ — 
Dj) in B ^ D^K^*^^ and B{D*^ L>°7r°) x B{D^ Dj) in B D*^K^, Br is the known 

^ K^^ 7r+7r-(K*o K+ir^) branching fraction in B ^ Z)(*)OkO(K*°), and ej is the signal 
reconstruction efficiency. We assume B{T{AS) B^B^) = 0.5. 

We measure 
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= (4.5 ± 1.9 ± 0.5) X 10" 
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= (6.2 ± 1.4 ± 0.6) X 10" 


-5 


B{B^ - 


> D^K*^) 


= (1.1 ± 1.1 ± 1.2) X 10" 


-5 



where the uncertainties are, respectively, statistical and systematic. For the decay B — > D K 
we use the Bayesian method to compute the upper limit Nul on the observed number of events 
at 90% confidence level as /g C{N) dN = 0.9, where C{N) is the binned maximum likelihood 
function from the fit to the ?ties distribution. We assume a fiat prior probability density function 
for B > 0. After accounting for the systematic uncertainties we obtain at 90% C.L. B{B^ —>■ 
WK*^) < 4.1 X 10""^ 
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In summary, we have presented evidence for the decay — > D*^K^ as well as new measure- 
ments of the branching fractions for the decays — > D^K^ and D^K*^ . Our measurements are 
in agreement with the expectation derived from Ref. [31 and with previous measurements ^Jj. We 
use the central value of our measurement for B{B^ — > D^K*^) and obtain r < 0.8 at the 90% C.L. 
from a central value of r = 0.4 it 0.2 (stat.) it 0.2 (syst.). The main contribution to the systematic 
uncertainty is from the estimated peaking background since most systematic uncertainties on the 
branching fractions cancel in the ratio. 
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Figure 2: Distribution of tues (left) and AE (right) for B D^*'i^K^ and 5° D^K*^ and 
D^K*^ candidates. The points are the data, the curve is the result of the fit, and the hatched 
histogram is the distribution of candidates in sidebands. 
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Figure 3: Distribution of cos 9h for selected D^K*^ candidates. The points are data, after 

subtracting the combinatorial background, and the curve is the result of a fit, which is consistent 
with the expected cos^ 6h distribution. 
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